Allow `enum` and `union` literals to also create SSA values #138759

scottmcm · 2025-03-20T18:41:17Z

Today, Some(x) always goes through an alloca, even in trivial cases where the niching means the constructor doesn't even change the value.

For example, https://rust.godbolt.org/z/6KG6PqoYz

pub fn demo(r: &i32) -> Option<&i32> {
    Some(r)
}

currently emits the IR

define align 4 ptr @demo(ptr align 4 %r) unnamed_addr {
start:
  %_0 = alloca [8 x i8], align 8
  store ptr %r, ptr %_0, align 8
  %0 = load ptr, ptr %_0, align 8
  ret ptr %0
}

but with this PR it becomes just

define align 4 ptr @demo(ptr align 4 %r) unnamed_addr {
start:
  ret ptr %r
}

(Of course the optimizer can clean that up, but it'd be nice if it didn't have to -- especially in debug where it doesn't run. This is like #123886, but that only handled non-simd structs -- this PR generalizes it to all non-simd ADTs.)

The commits:

The first is simplifying how the aggregate handling works. Past-me wrote something overly complicated, needing arrayvecs and zipping, depending on a careful iteration order of the fields, and fragile enough that even for just structs it needed extra double-checks to make sure it even made the right variant. It's replaced with something far more direct that works just like extract_field: use the offset to put it in exactly the correct immediate in the OperandValue. This doesn't support anything new, just refactors -- including moving some things off FunctionCx that had no reason to be there. (I have no idea why my past self put them there.)
The second extends that work to support more ADTs. That means handing variants other than FIRST_VARIANT, handling the active field for unions, refactoring the discriminant code so the Place and Operand parts can share the calculation, etc.
Apparently I can't just remove a dep without updating the lock file, oops.
I got annoyed at tag_field being usize, so changed it to FieldIdx

No need to build `ArrayVec`s; just put everything exactly where it goes.

rustbot · 2025-03-20T18:41:22Z

r? @Nadrieril

rustbot has assigned @Nadrieril.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2025-03-20T18:41:24Z

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

rustbot · 2025-03-20T18:58:00Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

saethlin · 2025-03-20T20:14:02Z

r? codegen

saethlin · 2025-03-20T20:14:16Z

Spiderman meme

scottmcm · 2025-03-20T20:22:43Z

@bors try @rust-timer queue

bors · 2025-03-20T20:23:55Z

⌛ Trying commit aa8ceb8 with merge 875f416...

Allow `enum` and `union` literals to also create SSA values Today, `Some(x)` always goes through an `alloca`, even in trivial cases where the niching means the constructor doesn't even change the value. For example, <https://rust.godbolt.org/z/6KG6PqoYz> ```rust pub fn demo(r: &i32) -> Option<&i32> { Some(r) } ``` currently emits the IR ```llvm define align 4 ptr `@demo(ptr` align 4 %r) unnamed_addr { start: %_0 = alloca [8 x i8], align 8 store ptr %r, ptr %_0, align 8 %0 = load ptr, ptr %_0, align 8 ret ptr %0 } ``` but with this PR it becomes just ```llvm define align 4 ptr `@demo(ptr` align 4 %r) unnamed_addr { start: ret ptr %r } ``` (Of course the optimizer can clean that up, but it'd be nice if it didn't have to -- especially in debug where it doesn't run. This is like rust-lang#123886, but that only handled non-simd `struct`s -- this PR generalizes it to all non-simd ADTs.) There's two commits you can review independently: 1. The first is simplifying how the aggregate handling works. Past-me wrote something overly complicated, needing arrayvecs and zipping, depending on a careful iteration order of the fields, and fragile enough that even for just structs it needed extra double-checks to make sure it even made the right variant. It's replaced with something far more direct that works just like `extract_field`: use the offset to put it in exactly the correct immediate in the `OperandValue`. This doesn't support anything new, just refactors -- including moving some things off `FunctionCx` that had no reason to be there. (I have no idea why my past self put them there.) 2. The second extends that work to support more ADTs. That means handing variants other than `FIRST_VARIANT`, handling the active field for unions, refactoring the discriminant code so the Place and Operand parts can share the calculation, etc.

bors · 2025-03-20T22:32:13Z

☀️ Try build successful - checks-actions
Build commit: 875f416 (875f416b9aa583f09153eac5b56d36fcd932b274)

rust-timer · 2025-03-20T23:49:26Z

Finished benchmarking commit (875f416): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	0.0%	[0.0%, 0.0%]	1
Regressions ❌ (secondary)	0.2%	[0.2%, 0.3%]	3
Improvements ✅ (primary)	-0.3%	[-0.5%, -0.2%]	7
Improvements ✅ (secondary)	-0.3%	[-0.4%, -0.3%]	2
All ❌✅ (primary)	-0.3%	[-0.5%, 0.0%]	8

Max RSS (memory usage)

Results (primary -2.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.3%	[-2.3%, -2.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-2.3%	[-2.3%, -2.3%]	1

Cycles

Results (secondary -2.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.1%	[-2.3%, -1.6%]	8
All ❌✅ (primary)	-	-	0

Binary size

Results (primary -0.1%, secondary -0.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.1%	[-0.2%, -0.0%]	57
Improvements ✅ (secondary)	-0.1%	[-0.3%, -0.0%]	27
All ❌✅ (primary)	-0.1%	[-0.2%, -0.0%]	57

Bootstrap: 775.297s -> 774.424s (-0.11%)
Artifact size: 365.52 MiB -> 365.52 MiB (0.00%)

rustbot · 2025-03-23T04:48:35Z

Some changes occurred to the CTFE machinery

cc @RalfJung, @oli-obk, @lcnr

Some changes occurred to the CTFE / Miri interpreter

cc @rust-lang/miri

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

It was already available as a generic parameter anyway, and it's not like we'll ever put a tag in the 5-billionth field.

scottmcm · 2025-03-28T21:49:10Z

Ben said I should re-roll this
r? codegen

workingjubilee · 2025-05-01T04:52:21Z

hello.

workingjubilee · 2025-05-01T05:40:59Z

ah, this is a "tomorrow" PR. poke me if I do not get to it then.

scottmcm · 2025-05-05T20:24:34Z

@workingjubilee Friendly ping, as requested.

workingjubilee · 2025-05-27T01:08:03Z

compiler/rustc_codegen_ssa/src/mir/operand.rs

+                // Real primitives only have one variant, but weird types like
+                // `Result<!, !>` turn out to also be "Primitive", and dead code
+                // like `Err(never)` needs to not ICE.
+                assert_eq!(v, FIRST_VARIANT);


cogitating...

I wonder about

enum Never<T> { Say(!), Again<T> }

The matter of what the "first" variant is probably gets funny when some are uninhabited...

workingjubilee · 2025-05-27T01:26:11Z

compiler/rustc_codegen_ssa/src/mir/operand.rs

+    pub(crate) fn supports_builder(layout: TyAndLayout<'tcx>) -> bool {
+        match layout.backend_repr {
+            BackendRepr::Memory { .. } if layout.is_zst() => true,
+            BackendRepr::Scalar(_) | BackendRepr::ScalarPair(_, _) => true,
+            _ => false,
+        }
+    }


is this builder used in so many places that this is simpler than just returning Result? or Option, I suppose

workingjubilee

some thoughts that occurred while slowly combing through

workingjubilee · 2025-05-27T01:29:08Z

compiler/rustc_codegen_ssa/src/mir/rvalue.rs

                    let layout = self.cx.spanned_layout_of(ty, span);
-                    !self.cx.is_backend_ref(layout)
+                    OperandRef::<Bx::Value>::supports_builder(layout)


I am not entirely sure why the logic was "isn't a ref" and is now "uses one of these representations"... was it relying on some upstream logical narrowing to e.g. not answer true on BackendRepr::SimdVector?

I suppose now that it's this it's more correct in any case, but still,

this probably isn't actually important, but unlike some of my other meandering comments I remain curious.

compiler/rustc_codegen_ssa/src/mir/place.rs

compiler/rustc_codegen_ssa/src/mir/operand.rs

workingjubilee

a nitty review, it seems, my condolences.

workingjubilee · 2025-05-29T22:43:55Z

compiler/rustc_codegen_ssa/src/mir/place.rs

+            if variant_index != untagged_variant {
+                let niche_layout = layout.field(cx, tag_field.as_usize());
+                let niche_llty = cx.immediate_backend_type(niche_layout);
+                let BackendRepr::Scalar(scalar) = niche_layout.backend_repr else {
+                    bug!("expected a scalar placeref for the niche");
+                };
+                // We are supposed to compute `niche_value.wrapping_add(niche_start)` wrapping
+                // around the `niche`'s type.
+                // The easiest way to do that is to do wrapping arithmetic on `u128` and then
+                // masking off any extra bits that occur because we did the arithmetic with too many bits.
+                let niche_value = variant_index.as_u32() - niche_variants.start().as_u32();
+                let niche_value = (niche_value as u128).wrapping_add(niche_start);
+                let niche_value = niche_value & niche_layout.size.unsigned_int_max();
+
+                let niche_llval = cx.scalar_to_backend(
+                    Scalar::from_uint(niche_value, niche_layout.size),
+                    scalar,
+                    niche_llty,
+                );
+                Some((tag_field, niche_llval))
+            } else {
+                None
+            }


note to self: mostly just a move from line 242

workingjubilee · 2025-05-29T23:09:00Z

compiler/rustc_codegen_ssa/src/mir/operand.rs

+            Err(s) if s.is_uninit_valid() => {
+                let bty = cx.type_from_scalar(s);
+                cx.const_undef(bty)
+            }


hm. for some reason this feels like it will bite my nose.

workingjubilee · 2025-05-29T23:29:04Z

compiler/rustc_codegen_ssa/src/mir/rvalue.rs

+            mir::Rvalue::Aggregate(ref kind, ref fields) => {
+                let (variant_index, active_field_index) = match **kind {
+                    mir::AggregateKind::Adt(_, variant_index, _, _, active_field_index) => {
+                        (variant_index, active_field_index)
+                    }
+                    _ => (FIRST_VARIANT, None),
+                };
+
                let ty = rvalue.ty(self.mir, self.cx.tcx());
                let ty = self.monomorphize(ty);
                let layout = self.cx.layout_of(ty);

-                // `rvalue_creates_operand` has arranged that we only get here if
-                // we can build the aggregate immediate from the field immediates.
-                let mut inputs = ArrayVec::<Bx::Value, 2>::new();
-                let mut input_scalars = ArrayVec::<abi::Scalar, 2>::new();
-                for field_idx in layout.fields.index_by_increasing_offset() {
-                    let field_idx = FieldIdx::from_usize(field_idx);
-                    let op = self.codegen_operand(bx, &fields[field_idx]);
-                    let values = op.val.immediates_or_place().left_or_else(|p| {
-                        bug!("Field {field_idx:?} is {p:?} making {layout:?}");
-                    });
-                    let scalars = self.value_kind(op.layout).scalars().unwrap();
-                    assert_eq!(values.len(), scalars.len());
-                    inputs.extend(values);
-                    input_scalars.extend(scalars);
+                let mut builder = OperandRef::builder(layout);
+                for (field_idx, field) in fields.iter_enumerated() {
+                    let op = self.codegen_operand(bx, field);
+                    let fi = active_field_index.unwrap_or(field_idx);
+                    builder.insert_field(bx, variant_index, fi, op);
                }

-                let output_scalars = self.value_kind(layout).scalars().unwrap();
-                itertools::izip!(&mut inputs, input_scalars, output_scalars).for_each(
-                    |(v, in_s, out_s)| {
-                        if in_s != out_s {
-                            // We have to be really careful about bool here, because
-                            // `(bool,)` stays i1 but `Cell<bool>` becomes i8.
-                            *v = bx.from_immediate(*v);
-                            *v = bx.to_immediate_scalar(*v, out_s);
+                let tag_result = codegen_tagged_field_value(self.cx, variant_index, layout);
+                match tag_result {
+                    Err(super::place::UninhabitedVariantError) => {
+                        // Like codegen_set_discr we use a sound abort, but could
+                        // potentially `unreachable` or just return the poison for
+                        // more optimizability, if that turns out to be helpful.
+                        bx.abort();
+                        let val = OperandValue::poison(bx, layout);
+                        OperandRef { val, layout }
+                    }
+                    Ok(maybe_tag_value) => {
+                        if let Some((tag_field, tag_imm)) = maybe_tag_value {
+                            builder.insert_imm(tag_field, tag_imm);
                        }
-                    },
-                );
-
-                let val = OperandValue::from_immediates(inputs);
-                assert!(
-                    val.is_expected_variant_for_type(self.cx, layout),
-                    "Made wrong variant {val:?} for type {layout:?}",
-                );
-                OperandRef { val, layout }
+                        builder.finalize(bx.cx())
+                    }
+                }
            }


I should re-review this a last time before approving

workingjubilee · 2025-05-29T23:39:49Z

compiler/rustc_codegen_ssa/src/mir/operand.rs

+        }
+    }
+
+    pub fn finalize(&self, cx: &impl CodegenMethods<'tcx, Value = V>) -> OperandRef<'tcx, V> {


...I'm sure this is a bit of a thick question to ask, but why is it called finalize exactly? It feels like it should get a documentation comment?

workingjubilee · 2025-05-29T23:48:46Z

compiler/rustc_codegen_ssa/src/mir/place.rs

+/// general, as it might be a niche value or have a different size.
+///
+/// It might also be an `Err` because the variant is uninhabited.
+pub(super) fn codegen_tagged_field_value<'tcx, V>(


mmh, tagged_field_value feels... off? I know it's an excruciatingly small detail, but codegen_tag_value sounds more correct... namely, like it doesn't codegen anything in the univariant case, which is important?

workingjubilee · 2025-05-29T23:57:50Z

@rustbot author

workingjubilee · 2025-05-30T00:13:38Z

On a more general note, it is always easier to review stuff when pure "move stuff around" things are in a separate commit from anything that actually changes logic.

scottmcm added 2 commits March 20, 2025 11:13

Simplify the aggregate-as-OperandRef path

5d13765

No need to build `ArrayVec`s; just put everything exactly where it goes.

Allow nearly all ADT aggregates to create operands

d69d324

rustbot assigned Nadrieril Mar 20, 2025

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Mar 20, 2025

This comment has been minimized.

Sign in to view

Reflect the removed dependencies in the cargo lock

aa8ceb8

rustbot assigned saethlin and unassigned Nadrieril Mar 20, 2025

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 20, 2025

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Mar 20, 2025

This comment has been minimized.

Sign in to view

Change tag_field to FieldIdx in Variants::Multiple

a9031aa

It was already available as a generic parameter anyway, and it's not like we'll ever put a tag in the 5-billionth field.

scottmcm force-pushed the operand-builder branch from 9423566 to a9031aa Compare March 23, 2025 05:04

scottmcm mentioned this pull request Mar 23, 2025

Simplify PartialOrd on tuples containing primitives #138135

Merged

rustbot assigned workingjubilee and unassigned saethlin Mar 28, 2025

scottmcm mentioned this pull request Apr 1, 2025

Rust fails to optimize away useless unwrap check #57166

Open

workingjubilee reviewed May 27, 2025

View reviewed changes

workingjubilee reviewed May 29, 2025

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 29, 2025

Allow enum and union literals to also create SSA values #138759

Are you sure you want to change the base?

Allow enum and union literals to also create SSA values #138759

Conversation

scottmcm commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Mar 20, 2025

Uh oh!

rustbot commented Mar 20, 2025

Uh oh!

This comment has been minimized.

rustbot commented Mar 20, 2025

Uh oh!

saethlin commented Mar 20, 2025

Uh oh!

saethlin commented Mar 20, 2025

Uh oh!

scottmcm commented Mar 20, 2025

Uh oh!

This comment has been minimized.

bors commented Mar 20, 2025

Uh oh!

bors commented Mar 20, 2025

Uh oh!

This comment has been minimized.

rust-timer commented Mar 20, 2025

Overall result: ❌✅ regressions and improvements - please read the text below

Uh oh!

rustbot commented Mar 23, 2025

Uh oh!

This comment has been minimized.

scottmcm commented Mar 28, 2025

Uh oh!

workingjubilee commented May 1, 2025

Uh oh!

workingjubilee commented May 1, 2025

Uh oh!

scottmcm commented May 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

workingjubilee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

workingjubilee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

workingjubilee commented May 29, 2025

Uh oh!

workingjubilee commented May 30, 2025

Uh oh!

Uh oh!

Allow `enum` and `union` literals to also create SSA values #138759

Allow `enum` and `union` literals to also create SSA values #138759

scottmcm commented Mar 20, 2025 •

edited

Loading